Future directions in learning to rank

The results of the learning to rank challenge showed that the quality of the predictions from the top competitors are very close from each other. This raises a question: is learning to rank a solved problem? On the on hand, it is likely that only small incremental progress can be made in the “core” and traditional problematics of learning to rank. The challenge was set in this standard learning to rank scenario: optimize a ranking measure on a test set. But on the other hand, there are a lot of related questions and settings in learning to rank that have not been yet fully explored. We review some of them in this paper and hope that researchers interested in learning to rank will try to answer these challenging and exciting research questions. 1. Learning Theory for Ranking Many learning to rank algorithms have been shown effective through benchmark experiments. However, sometimes benchmark experiments are not as reliable as expected due to the small scales of the training and test data. In this situation, a theory is needed to guarantee the performance of an algorithm on infinite unseen data. Statistical learning theory, specifically the generalization theory, investigates the bound between the risk on the finite training data and the risk on infinite test data. In learning to rank, the risk on the training data is defined with a surrogate loss function (e.g., the pairwise losses in Ranking SVM (Joachims, 2002) and RankBoost (Freund et al., 2003)), while the risk on the test set is measured by a ranking measure (e.g., 1-NDCG or 1-MAP). Therefore, to obtain a generalization bound in this setting, we need to address the following issues: (i) a reasonable assumption on the data generation (e.g., queries and documents), (ii) a generalization bound regarding the surrogate loss function; (iii) the relationship between the surrogate loss function and the ranking measure; (iv) and the existence of the limit of the ranking measure when the number of documents approaches infinity. As for these issues, there have been a number of attempts but still a large open space to explore. • Assumption on data generation. In (Agarwal and Niyogi, 2005; Clemencon and Vayatis, 2007) it is assumed that the documents in the training data are sampled in an i.i.d. manner, no matter which queries they are associated with. However, it is c © 2011 O. Chapelle, Y. Chang & T.-Y. Liu.